Merged
Conversation
feat(xatu_public_contributors): implement service for public data
parithosh
approved these changes
Apr 2, 2025
Member
|
LGTM |
This commit introduces a new script, `create_new_proto_file.sh`, that automates the creation of new protocol buffer files and directories. The script takes a proto name as an argument, creates the corresponding directory and `.proto` file, and updates the `buf.work.yaml` file to include the new directory. This streamlines the process of adding new proto definitions to the project.
feat: add storage compression feat: add storage store params and unified store method feat: add xatu public contributors service This commit introduces several enhancements and fixes: - Adds integration tests for ClickHouse client using testcontainers. - Implements storage compression using gzip algorithm. - Introduces StoreParams struct and a unified Store method in the storage package to handle encoding, compression, atomicity, content type, and metadata. - Implements the Xatu Public Contributors service. - Fixes and improves the beacon chain timings service. - Updates the beacon slots service. - Updates the xatu public contributors service. - Adds .gitignore entry for debug_bin
This commit refactors the storage client to accept a context for cancellation and deadlines. It also adds compression to the stored data to reduce storage costs and improve performance. The following changes were made: - Updated the `storage.Get` and `storage.Store` methods to accept a context. - Added compression to the `storage.Store` method. - Updated all services to use the new `storage.Get` and `storage.Store` methods. - Removed bucket creation logic from storage client and fail if bucket does not exist. - Removed content type detection logic from storage client and rely on codec.
feat(beacon_slots.go): implement beacon slots gRPC service feat(beacon_slots.go): implement beacon slots service feat: add data fetching logic for beacon slots This commit introduces the beacon slots service, including its configuration, gRPC handler, and core logic. It also includes data fetching from ClickHouse and transformation for storage. The changes include: - Added `modules` config to `config.go` - Implemented the `BeaconSlotsHandler` in `grpc/beacon_slots.go` - Implemented the `BeaconSlots` service in `internal/service/beacon_slots/beacon_slots.go` - Added data fetching logic in `internal/service/beacon_slots/data_fetching.go` - Added transformation logic in `internal/service/beacon_slots/transform.go` - Added processing logic in `internal/service/beacon_slots/processing.go` - Removed `json.go` feat: add BlockArrivalTime, BlobArrivalTime, BlobArrivalTimes messages feat: add FullTimings message to include metadata and geo location feat: add SlimTimings message to drop metadata and geo location feat: update BeaconSlotData message to use SlimTimings feat: add BeaconSlots service definition refactor: remove BeaconSlotsService and related code refactor: rename Timings to SlimTimings refactor: update beacon_slots.proto to include new messages
feat(go.mod): add ethwallclock dependency feat(go.mod): add prysmaticlabs/go-bitfield dependency feat(go.mod): remove durafmt dependency feat(pkg/internal/lab/ethereum): add config struct for ethereum feat(pkg/internal/lab/ethereum): add network struct for ethereum feat(pkg/internal/lab/state): add state management package feat(pkg/internal/lab/storage): add exists method to storage client feat(pkg/server/internal/service/beacon_chain_timings): refactor state feat(pkg/server/internal/service/beacon_chain_timings): add state adapter feat(pkg/server/internal/service/beacon_slots): refactor beacon slots feat(pkg/server): refactor server to use new ethereum client
feat: remove xatuclickhouse generated files This commit removes the generated files for the xatuclickhouse package. The processing logic was changed to be done in the server side. feat(processing.go): process slots data in parallel This commit introduces parallel processing of slot data to reduce latency. It uses errgroup to run independent data fetching tasks concurrently, improving overall performance. ``` chore: remove unused xatuclickhouse models This commit removes a large number of unused models from the `pkg/xatuclickhouse/models` directory. These models were autogenerated but are no longer used in the project. Removing them cleans up the codebase and reduces the potential for confusion. feat: remove mev_relay_validator_registration model and ptr utils The mev_relay_validator_registration model and ptr utils are removed because they are no longer needed.
This commit significantly updates the README to accurately describe the current Go-based backend architecture, replacing the outdated Python implementation details. It includes details on the SRV and API services, key backend modules, technologies used, and updated development instructions. The multi-network support feature description is also updated.
feat: introduce config API and related data structures This commit introduces the config API, which allows retrieving configuration information for the lab. The commit also defines the data structures used by the config API, including: - ConfigResponse: The main response message for the config API. - ModuleConfig: Configuration for a specific module. - TimeWindow: A time window for a module. - EthereumConfig: Configuration for Ethereum. - EthNetworkConfig: Configuration for an Ethereum network. - Forks: Fork configuration for an Ethereum network. - ForkDetail: Details about a specific fork. build: regenerate lab_api.pb.go This commit updates the lab_api.pb.go file by regenerating it. This ensures that the generated code is up-to-date with the latest proto definition. feat: add GetFrontendConfig endpoint to lab API This commit introduces a new `GetFrontendConfig` endpoint to the Lab API. This endpoint retrieves the frontend configuration, allowing clients to dynamically adjust their behavior based on the server's configuration. The changes include: - Added the `GetFrontendConfig` method to the `LabService` in `pkg/server/internal/grpc/lab.go`. - Implemented the `GetFrontendConfig` method in the `pkg/api/service.go` to handle the new endpoint. - Added the corresponding protobuf definitions in `pkg/api/proto/lab_api.proto` and updated the generated code. - Modified the `pkg/server/internal/service/lab/lab.go` to implement the logic for retrieving the frontend configuration. - Updated the `pkg/server/internal/service/beacon_slots/beacon_slots.go` to include the `FrontendModuleConfig` method. - Updated the `pkg/server/internal/service/beacon_chain_timings/beacon_chain_timings.go` to include the `FrontendModuleConfig` method. - Updated the `pkg/server/internal/service/xatu_public_contributors/xatu_public_contributors.go` to include the `FrontendModuleConfig` method. - Removed the `pkg/server/internal/service/lab.go` file. feat(proto): add GetConfigResponse message and module config feat(server): add lab service and update dependencies This commit introduces a new GetConfigResponse message in the protocol buffer definition, which includes module configurations and Ethereum settings. It also adds a new lab service that depends on the Ethereum client, cache client, beacon chain timings, Xatu public contributors, and beacon slots services. The server initialization logic is updated to include the new lab service. The stop function is updated to stop all services gracefully.
This commit improves the service's shutdown process by using a cancelable context and WaitGroup to ensure all components stop gracefully. It also adds a timeout to prevent indefinite blocking during shutdown and logs warnings if the timeout is exceeded. Additionally, the commit passes the service context to dependency initialization functions.
This change ensures that the process context is derived from the context passed to the Start method, allowing for better control and propagation of cancellation signals.
The backend directory and its contents have been removed. This includes the Dockerfile, configuration files, Python scripts, and related modules. The removal is part of a larger refactoring effort to streamline the project and potentially integrate backend functionalities elsewhere or rely on external services. feat: add metrics instrumentation to cache, clickhouse, ethereum, and geolocation services This commit introduces metrics instrumentation to the cache, clickhouse, ethereum, and geolocation services using the prometheus client library. The changes include: - Added metrics collection for cache operations (get, set, delete) with hit/miss/error status, operation duration, and number of items in the cache. - Added metrics collection for ClickHouse queries (query, query_row, exec) with success/error status and query duration. - Added metrics collection for Ethereum network wallclock and configuration loading. - Added metrics collection for geolocation lookups with success/error/not_found status, lookup duration, and database load operations. The metrics are exposed through Prometheus and can be used to monitor the performance and health of the services. feat(leader): add metrics to leader election process feat(geolocation): add metrics service to geolocation feat(locker): add metrics to distributed lock implementation feat(state): add metrics to state management feat(storage): add metrics to storage operations feat(xatu): add metrics to xatu client feat(beacon_chain_timings): add metrics to beacon chain timings service feat(beacon_slots): add metrics to beacon slots service feat(lab): add metrics to lab service feat(xatu_public_contributors): add metrics to xatu public contributors service feat: add metrics endpoint and HTTP server This commit introduces a new HTTP server that exposes Prometheus metrics at the `/metrics` endpoint. The server runs on port 9090 by default. The following changes were made: - Added a new `httpServer` field to the `Service` struct. - Added a new `startMetricsServer` method to start the HTTP server. - Modified the `Start` method to start the HTTP server. - Modified the `stop` method to gracefully shut down the HTTP server. - Initialized metrics service as a dependency. - Passed metrics service to all services.
The metricsCollector is initialized during service startup. Therefore, nil checks before using the collector are unnecessary and have been removed to simplify the code.
Introduces new Prometheus metrics to track the state and age of processed data within the `beacon_chain_timings`, `beacon_slots`, and `xatu_public_contributors` services. - `beacon_chain_timings`: Adds `state_last_processed_seconds` and `state_age_seconds` to track the last processed time and age of block timings and CDF data per network and window. - `beacon_slots`: Adds `state_last_processed_slot` and `state_slot_age` to track the last processed slot and its age (in slots) for each processor (head, trailing, backfill) per network. - `xatu_public_contributors`: Adds `state_last_processed_seconds`, `state_window_last_processed_seconds`, `state_age_seconds`, and `state_window_age_seconds` to track the last processed time and age for each processor and its specific time windows per network. These metrics provide better visibility into the freshness and processing status of the data managed by these services.
…ocessing fix(beacon_slots): increase leader refresh interval to reduce etcd load
feat(api): enable xatu_public_contributors module by default fix(frontend): update vite proxy target port to 8080
…upport flat and nested formats refactor(beacon_slots): update SQL query to include geo and time data for block events
feat(geolocation): add support for 'coordinates' header in CSV download
fix(ci): update component name from backend to pkg in containers workflow
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.